Distributed rendering and display system

ABSTRACT

A distributed rendering and display system comprises a host device for rendering z-buffers from a scene, a set of pipeline rendering devices, and a set of display devices. Each pipeline rendering device is remotely connected to the host device. Each display device is remotely connected to a pipeline rendering device in the set of pipeline rendering devices. Each rendered z-buffer from the scene is associated with a display view perspective of the scene. Each display device in the set of display devices provides display view perspectives of the scene to one or more users.

CROSS-REFERENCE TO RELATED PATENT APPLICATIONS

This patent application is a continuation application of U.S. patentapplication Ser. No. 17/549,614, filed on Dec. 13, 2021, which claimspriority to U.S. Provisional Patent Application Ser. No. 63/125,393,filed on Dec. 14, 2020, each of which is hereby incorporated byreference in its entirety.

TECHNICAL FIELD

This disclosure relates to rendering and more particularly to renderingof a three-dimensional scene.

BACKGROUND

Current real-time rendering architectures typically require that anyuser viewing a rendered scene must do so using rendering hardware toperform both culling on the scene geometry and rendering of a framebuffer for a computer-generated (CG) camera. This means users viewing ascene with complex and/or extensive geometry require consuming highresource and bandwidth in the rendering hardware.

DESCRIPTION OF DRAWINGS

FIG. 1A is a schematic of a host device and multiple rendering pipelinesconfigured to display a view of a scene rendered on the host device.

FIG. 1B is a schematic of a host device remotely connected to multiplerendering pipelines configured for displaying a view of a scene renderedon the host device.

FIG. 1C is a schematic of a host device and multiple rendering devicesthat are remotely connected to multiple display devices for displaying aview of a scene rendered on the host device.

FIG. 2 is an illustration of a CG 3D scene and the perspectives fromwhich multiple users are viewing the scene.

FIG. 3 is an illustration of a memory buffer that is partitioned intomultiple memory blocks for storing unique z-buffers, each of which isassigned to a unique rendering pipeline.

FIG. 4 is an illustration of a z-buffer received by a rendering deviceand rendered to a frame buffer to be displayed on a display device.

FIG. 5 is an illustration of a camera configuration used to render a CGscene.

FIG. 6 is an illustration of a camera configuration wherein a single CG“multiview camera” renders a scene from multiple perspectives.

Like reference symbols in the various figures indicate like elements.

DETAILED DESCRIPTION

The specification describes systems, methods, devices, and techniquesfor distributed rendering and displaying of scene data. For example, adistributed rendering and display system includes a host device, framerendering devices (the “pipeline rendering devices”) connected to thehost device, and display devices associated with the frame renderingdevices.

The host device receives scene data from an external source, and userdata and computer-generated (CG) camera definition data from each of therendering devices. The host device includes a host rendering subsystemand a memory buffer having memory blocks associated with the pipelinerendering devices. The host rendering subsystem receives the scene data,the user data, and the CG camera definition to cull scene geometry andrender z-buffers that are stored in the memory blocks associated withthe pipeline rendering devices. Each of the pipeline rendering devicesreceive the z-buffers associated with the pipeline rendering device torender and provide pipeline frame buffers that reflect a viewperspective of the scene data at the display device associated with thepipeline rendering device. For example, each of the display devices maysimultaneously display the scene data having a view perspective that isdetermined by the pipeline frame data that was rendered by the pipelinerendering device associated with the display device.

An example process for rendering frame buffers for the display devicesincludes: a) each rendering pipeline provides the host device with acomputer-generated (CG) camera definition; b) a memory buffer is createdon the host device to hold the rendered z-buffers for each of therendering pipelines; c) the CG camera definitions are used to renderscene geometry to a unique z-buffer for each rendering pipeline; d) eachz-buffer is transmitted to its associated rendering pipeline; e) eachrendering pipeline uses the received z-buffer to render an appropriateframe buffer; and f) the rendered frame buffer is sent to the associateddisplay device.

In other embodiments, the rendering and display system may include aplurality of host devices associated with the pipeline renderingdevices. The distributed rendering and display system may include thehost device and the rendering pipeline devices being disposed on remotemachines. Alternatively, the host device and the rendering pipelinedevice may be disposed in the same box (i.e., computer). Also, the hostdevice and rendering device may be structured together such that thedisplay devices are remotely connected to the rendering device.

For example, selecting the z-buffer may depend on the type of displaydevice. In a single display, the z-buffer may be the frustum of a singleview. In another example, a display device with more than one viewfrustum may have all the views in one z-buffer. As the number of theviews increase, the size and data contained in a z-buffer may alsoincrease while the number of z-buffers remain the same. According toanother embodiment, the z-buffers may be selected for each frame whenthere's no difference between the previous frame's camera position andnext position.

The rendering structures and processes are distributed between the hostdevice for cull scene geometry and z-buffer rendering and the pipelinerendering devices for their associated frame buffer rendering. The hostdevice is configured to store elements of the scene into an accelerationstructure and create multiple display device specific z-buffers. Eachpipeline rendering device may be structured to only have enoughrendering power to use the received z-buffer to render a frame buffer ofresolution equal to the resolution of its associated display device.This supports controlling the rendering (or “processing”) power requiredof each rendering pipeline to be reduced and independent from thecomplexity of the scene. Also, this can increase speed for transferringlarge amounts of data (the z-buffer) to multiple boxes (the renderingpipeline devices). The system can support displaying N number of viewsfor the computational resource cost of displaying one view.

In another embodiment, a distributed rendering and display system forproviding display view perspectives of a scene includes a host devicefor rendering z-buffers from the scene; a set of pipeline renderingdevices; and a set of display devices. Each display device in the set ofdisplay devices is associated with a pipeline rendering device in theset of pipeline rendering devices. The pipeline rendering devicegenerates user input data and computer-generated camera definitionassociated with the display device. The host device receives the userinput data and computer-generated camera definition to select a set ofz-buffers from the rendered z-buffers. The pipeline rendering devicereceives the set of z-buffers to render frame buffer. The display devicereceives the rendered frame buffers to determine the display viewperspective for the scene. The host device may include a host renderingsub-system for rendering the z-buffers and a memory buffer having a setof memory blocks for storing the rendered z-buffers. The pipelinerendering device may be associated with a memory block from the group ofmemory blocks. Each of the display devices may simultaneously displaythe scene having a view perspective that is determined by the framebuffers rendered by the pipeline rendering device associated with thedisplay device. The host device may be directly connected to the set ofpipeline rendering devices and set of display devices. Alternatively,the host device may be remotely connected to the set of pipelinerendering devices and set of display devices. According to anotheraspect, the host device and the set of pipeline rendering devices may beremotely connected to the set of display devices.

According to another aspect, a rendering system includes a host devicehaving a memory buffer for storing z-buffers; a rendering device; and adisplay device. The host device renders the scene to the z-buffer, therendering device receives the z-buffer to render a frame buffer, and thedisplay device displays the frame buffer.

According to another embodiment, a distributed rendering and displaysystem that provides display view perspectives of a scene. Thedistributed rendering and display system includes a host device forgenerating z-buffers from the scene; a pipeline rendering device; and adisplay device. The pipeline rendering device generates user input dataand computer-generated camera definition associated with the displaydevice. The host device receives the user input data andcomputer-generated camera definition to select a set of z-buffers fromthe generated z-buffers; the pipeline rendering device receives the setof z-buffers to render frame buffers; and the display device receivingthe frame buffers to determine the display view perspective for thescene.

This system can also support real-time rendering, collaboration betweenmultiple users, and enables each rendering pipeline to use numeroustypes of desired displays, including but not limited to: virtual reality(VR), augmented reality (AR), mixed reality (MR), computer monitor,smartphone screen, television screen, autostereoscopic,three-dimensional (3D), glasses-free 3D, lenticular screen, holographic,light field, and volumetric display. Also, standard graphics structuremay be used on the host rendering subsystem and on the pipelinerendering devices.

FIG. 1A is an example embodiment of a distributed rendering and displaysystem 1000 comprising a host device 1010 disposed in communication withat least one rendering pipeline 1030. Additional rendering pipelines maybe disposed in communication with host device 1010, represented byrendering pipelines 1120, 1130. Rendering Pipeline 1030 includes arendering device 1150 disposed in communication with a display device1100. Display device 1100 may be configured to provide a user with aview of 3D, or higher dimensionality, CG scene (“Scene”) 2000 (shown inFIG. 2 ). Host Device 1010 and rendering device 1150 shareresponsibility for rendering a view of scene 2000. In some embodiments,multiple rendering pipelines, exemplified by rendering pipelines 1030,1120, 1130, may be connected to at least one host device 1010, whereinthe users of each rendering pipeline may view scene 2000 from a uniqueperspective. In some embodiments, display device 1100 may be configuredto provide a user with a view of a real-world scene captured withcameras. The embodiments for a camera include a computer-generatedcamera, a virtual camera, a synthetic camera, a physical camera, or areal camera.

Host device 1010 may be configured to receive data via Input 1020 fromone or more external sources, which may include without limitation oneor more of the following list: internet, intranet, user input, or sensorinput, physical cameras and external scene data such as satellite/radar.Host device 1010 may be further configured to receive CG cameradefinition 1050 from render pipeline 1030. CG camera definition 1050 mayinclude without limitation one or more of the following cameraspecifications: position, orientation, field of view, view frustum, andnear and far clipping planes. Host Device 1010 may use CG cameradefinition 1050 to render the geometry of a scene 2000 to Z-buffer 1060,which may be sent to render pipeline 1030. Device 1010 may be stillfurther configured to receive user input 1040 from render pipeline 1030.

Rendering device 1150 may be configured to receive (or utilize) Z-buffer1060 from host device 1010 to render frame buffer 4100 (see FIG. 4 ).Frame Buffer 4100 may then be displayed on display device 1100. In someembodiments, Rendering device 1150 may be a full, stand-alone, computersystem. In other embodiments, rendering device 1150 may be separateprocessing elements including but not limited to CPU, GPU, FPGA andASIC. In still further embodiments, Rendering device 1150 may beincorporated, integrated or embedded into display device 1100. Renderingdevice 1150 may communicate with host device 1010 via a local wirednetwork, local wireless network, or network connection over theinternet. In other embodiments, host device 1010 and rendering device1150 may both be part of a single computer in which the two devices maybe disposed in communication with each other using the system bus. Insome embodiments, display device 1100 and rendering device 1150 may beconnected via a local wired or wireless connection. In otherembodiments, display device 1100 may be connected to rendering device1150 via an internet or intranet connection.

Display device 1100 may be of a type that includes without limitationone or more of the following list: VR, AR, MR, computer monitor,smartphone screen, television screen, autostereoscopic, 3D, glasses-free3D, lenticular screen, holographic, light field, volumetric, or anyother standard or non-standard display type. Display Device 1100 mayprovide rendering device 1150 with CG camera definition 1050. Positionand orientation characteristics of CG Camera Definition 1050 may bedetermined using sensors connected to or integrated with display device1100. In some embodiments, display device 1100 may be a head wornsystem, and position and orientation characteristics of CG cameradefinition 1050 may be determined by sensors integrated into displaydevice 1100. In other embodiments, display device 1100 may include userinput devices or methods that are capable of determining the positionand orientation characteristics of CG camera definition 1050. Displaydevice 1100 may additionally provide rendering device 1150 with field ofview, view frustum, and near and far clipping planes characteristics ofCG camera definition 1050.

FIG. 1B is an example of the distributed rendering and display system100 of FIG. 1A with a host device being remotely connected 1210 tomultiple rendering pipelines configured to display a view of a scenerendered on the host device. In this example, the rendering pipeline isremotely assisted by the host device.

FIG. 1C is an example of the distributed rendering and display system100 of FIG. 1A with a host device and multiple rendering devices thatare remotely connected 1220 to multiple display devices to display aview of a scene rendered on the host device.

FIG. 2 is an illustration CG scene 2000 that is rendered on host device1010 (shown in FIG. 1 ). Scene 2000 may contain 3D objects and lightsources exemplified by 3D object 2100, and light source 2300. Scene 2000may be bounded by scene boundary 2200. Scene 2000 may be viewed by anumber of users (not shown). For display devices 1100 intended for useby a single user, the number of users may be substantially equal to thenumber of rendering pipelines connected to host device 1010 (shown inFIG. 1 ). Likewise, for display devices 1100 intended for use bymultiple users, the number of users may be substantially different thanthe number of rendering pipelines connected to host device 1010. Eachuser may view scene 2000 from a unique perspective, represented by userperspective 2400. User perspective 2400 may represent the location andorientation of a CG camera used to render the geometry of scene 2000 toz-buffer 1060 (shown in FIG. 1 ). A unique z-buffer may be rendered foreach rendering pipeline.

FIG. 3 is an illustration of memory buffer 3000 on host device 1010(shown in FIG. 1 ). Memory buffer 3000 may be created by rendering thegeometry of scene 2000 (shown in FIG. 2 ) using multiple CG cameras (notshown). Memory buffer 3000 may be partitioned into a number of memoryblocks 1060, 3010, 3020 for storing Z-buffers associated with renderingpipelines 1030, 1120, and 1130.

Unique CG cameras may exist for each rendering pipeline 1030, 1120,1130, and be used to render scene 2000 to Z-buffers 1060, 3010, 3020respectively. Once rendered, Z-buffers 1060, 3010, 3020 may betransmitted to the rendering device associated with rendering pipelines1030, 1120, 1130, respectively.

The size of memory buffer 3000 may be determined during startup of therendering application on host device 1010 (shown in FIG. 1 ). Forexample, the size of memory buffer 3000 may be specified by the size ofz-buffers 3100, 3200, 3300 via communication with rendering pipelines1030, 1120, 1130 (shown in FIG. 1 ). If additional rendering pipelinesare connected to host device 1010 while the system is running, memorybuffer 3000 may dynamically increase in size to accommodate the newrendering pipelines. If a rendering pipeline is disconnected from hostdevice 1010 while running, memory buffer 3000 may remain the same sizeand a portion of said memory buffer 3000 may be unused. Memory buffer3000 may dynamically decrease in size. Alternatively, memory buffer 300remains the same size when there is sufficient memory on host device1010.

FIG. 4 is an illustration of z-buffer 1060, received by rendering device1150 (shown in FIG. 1 ), and frame buffer 4100, rendered by renderingdevice 1150. Frame buffer 4100 may be rendered by rendering device 1150using the standard graphics pipeline. In some embodiments, frame buffer4100 may be rendered using a fragment shader. In some embodiments,z-buffer 1060 may include multiple views of scene 2000. In someembodiments, CG camera definition 1050 may define the location ofmultiple individual CG cameras as shown by camera configuration 5600(shown in FIG. 5 ). In some embodiments, z-buffer 1060 may be split intosections wherein each section represents a unique view of scene 2000,exemplified by views 4200, 4300, 4400, 4500. Each of views 4200, 4300,4400, 4500 may be from positions which are distinct from one another butsubstantially similar to user perspective 2400 (shown in FIG. 2 ).z-buffer 1060 may comprise texels. Frame buffer 4100 may be created byrearranging the texels of Z-buffer 1060. For example, information fromtexel 4220 in Z-buffer 1060 may be copied to texel 4700 in frame buffer4100. Frame buffer 4100 may then be sent to the pixel array of displaydevice 1100.

The locations to which texels are rearranged between Z-buffer 1060 andframe buffer 4100 may depend on the configuration of display device 1100(shown in FIG. 1 ). For example, display device 1100 may be configuredas a multiview display wherein a user perceives a different view basedon their position relative to the display. In this manner, when framebuffer 4100 is displayed on the pixel array of display device 1100, auser may see view 4200 when located at a first position (view zone 1)relative to display device 1100, view 4300 when at a second position(view zone 2), view 4400 when in a third position (view zone 3), andview 4500 when in a fourth position (view zone 4). In furtherembodiments, display device 1100 may be configured as a volumetricdisplay wherein each view appears at a different depth location withinthe display. In still further embodiments, display 1100 may beconfigured as an AR or VR headset wherein unique views are presented tothe user's right and left eyes. In some embodiments, the rearrangementof texels between z-buffer 1060 and frame buffer 4100 may include imagedistortion to compensate for optical aberrations in the display device1100, such as geometric distortion due to the lenses of an AR or VRdevice. The number of distinct views present in z-buffer 1060 and framebuffer 4100 may be defined by CG camera definition 1050 (shown in FIG. 1) and may differ based on the type or configuration of display device1100. In some embodiments, each display device 1100 may adhere to astandard which defines how the number of views are reported to renderingdevice 1150 and how rendering device 1150 may create CG cameradefinition 1050. In further embodiments, CG camera definition 1050 maybe used to determine the way in which texels are rearranged betweenz-buffer 1060 and frame buffer 4100.

In some embodiments, optimizations may be made to reduce the memory sizeof z-buffer 1060 and thus reduce the bandwidth required to transferz-buffer 1060 from host device 1010 to rendering pipeline 1030. Oneexample of an optimization is as follows: Rather than rendering eachview of scene 2000 to different regions of z-buffer 1060 (represented byviews 4200, 4300, 4400, 4500), all views may be rendered into anotherz-buffer having a smaller number of total texels than z-buffer 1060, butin which overlap exists between different views. In this fashion, eachtexel may represent a unique location within scene 2000. However, somelocations within scene 2000 may be visible from multiple views. Thecolor, hue, and intensity of a single texel may be determined byrendering the color, hue, and intensity of the location within scene2000 associated with said texel from all views to which it is visibleand averaging the results. In some embodiments, an equation may be usedto determine the color, hue, and intensity of a texel associated with aparticular location in scene 2000, and the contributions from each viewto which said location is visible may each be given a weight. Theweighting of any view may be equal or not equal to the weighting of anyother view.

FIG. 5 is an illustration of scene 2000 and camera configuration 5600configured to render views 4200, 4300, 4400, 4500 of Scene 2000 toz-buffer 1060 (shown in FIG. 4 ). While four views are shown by way ofexample, camera configuration 5600 may be configured to render generallyany number of views. In some embodiments, camera configuration 5600 mayinclude multiple standard CG cameras exemplified by camera 5200, 5300,5400, 5500, to render views 4200, 4300, 4400, 4500 respectively.

Rendering device 1150 may be configured to render views 4200, 4300,4400, 4500 sequentially or in parallel. In some embodiments, the overallrendering speed may be improved by rendering views 4200, 4300, 4400,4500 in parallel as this may be faster than rendering sequentially. Insome embodiments, views 4200, 4300, 4400, 4500 may be renderedsequentially using a standard graphics pipeline. In other embodiments,views 4200, 4300, 4400, 4500 may be rendered in parallel using a customgraphics pipeline. A custom graphics pipeline may be implemented using alow-level Graphics Processing Unit (“GPU”) language including but notlimited to Vulkan, Metal, or DirectX 12 or higher. A custom graphicspipeline may enable one or more processing units (“Processing Unit”) ofthe processor of rendering device 1150 to be assigned to render each ofviews 4200, 4300, 4400, 4500. The processor of rendering device 1150 maybe of the type including but not limited to CPU, GPU, FPGA and ASIC.Views 4200, 4300, 4400, 4500 may be rendered using a raster-based or raytracing-based method. In some embodiments, a parallel distributedrendering method may be employed wherein multiple Processing Units ofthe GPU may share responsibility of rendering a particular view. Inother embodiments, Processing Units from multiple GPUs may be usedtogether to perform parallel distributed rendering for a single view,wherein the GPUs are all part of the same computer system. In furtherembodiments, the multiple GPUs used for parallel distributed renderingmay not be in the same computer system and are instead connected using amethod including but not limited to: local network, internet, or otherstandard or non-standard method. In still further embodiments,Processing Units used for parallel distributed rendering may not be partof a GPU but instead part of a CPU, FPGA, ASIC, or other type ofprocessor.

FIG. 6 is an illustration of a multiview camera configuration 6600,which is an alternative to camera configuration 5600 (shown in FIG. 5 ).Multiview camera configuration 6600 is configured as a single multiviewcamera, rather than multiple standard CG cameras which is the typicalcase for camera configuration 5600. Multiview camera configuration 6600may enable z-buffer 1060 to be rendered in a single render pass,regardless of the number of views. By contrast, most current GPUhardware is limited to a maximum of 16 camera views per render pass.Thus, for large numbers of views, current GPU hardware requires multiplerender passes to render z-buffer 1060. Multiview camera configuration6600 may be primarily intended to be used in conjunction withray-tracing-based rendering methods, however it may be used withmodified raster-based rendering methods as well. One of the steps ofraster-based rendering is known as z-buffering, a process that sharessimilarities with ray-tracing. During the z-buffering process, a CGcamera with a defined texel resolution emits a ray for each texel todetermine the depth location at which that ray intersects an object inthe scene. The location at which the projected ray intersects with theobject defines the z-buffer value of the texel associated with the ray.This z-buffer information may be used to create z-buffer 1060.

The first step to defining the structure of multiview cameraconfiguration 6600 is to determine the desired layout of the texels inz-buffer 1060. In some embodiments, there may be no difference betweenZ-buffer 1060 and frame buffer 4100 (shown in FIG. 4 ), in which caseZ-buffer 1060 may be sent directly to the screen of display device 1100.

Some display devices such as multiview displays and light field displaysemit directional light when a pixel is illuminated. For these types ofdisplays, CG camera definition 1050 may include directional informationabout the ray of light associated with each pixel of display device1100. For each texel of Z-buffer 1060, a virtual texel may be definedwithin multiview camera configuration 6600 on virtual texel plane 6000,exemplified by virtual texel 6100, 6120, 6140. A ray may be defined foreach virtual texel which originates at a Ray Origin Position (“ROP”) andpasses through the center of said virtual texel. Rays 6300, 6320, 6340may originate at ROP 6200, 6220, 6240 respectively, and pass through thecenter of virtual texels 6100, 6120, 6140 respectively. In someembodiments ROP 6200, 6220, 6240 may be selected such that if virtualtexel plane 6000 were scaled to have the same size as the display ofmultiview or light field display device 1100 and superimposed over saiddisplay, rays 6300, 6320, 6340 may be substantially collinear with therays of light emitted by said display device when the physical pixelsassociated with virtual texels 6100, 6120, 6140 are illuminated. Nearclipping plane 6500 and far clipping plane 6600 may bound the volumewithin which initial ray intersections are detected. Standard raytracing methods may be used to determine the color and shading of thetexels of Z-buffer 1060 associated with each ray.

The term “N number of views”, or any other term similarly relating to anumber of views, may be used synonymously with “multiple views,” “one ormore views,” or “any number of views.”

A number of example embodiments have been described. Nevertheless, itwill be understood that various modifications may be made withoutdeparting from the spirit and scope of the devices and methods describedherein.

What is claimed is:
 1. A distributed rendering and display system for providing display view perspectives of a scene, the distributed rendering and display system comprising: a host device for rendering z-buffers from the scene; a set of pipeline rendering devices; and a set of display devices; wherein; each pipeline rendering device in the set of pipeline rendering devices is remotely connected to the host device; each display device in the set of display devices is remotely connected to a pipeline rendering device in the set of pipeline rendering devices; each rendered z-buffer from the scene is associated with a display view perspective of the scene; and each display device in the set of display devices provides display view perspectives of a scene to one or more users.
 2. A distributed rendering and display system for providing display view perspectives of a scene, the distributed rendering and display system comprising: a host device for rendering z-buffers from the scene; a set of pipeline rendering devices; a set of display devices; at least one computer-generated (CG) camera; and at least one CG camera definition; wherein; the host device is remotely connected to one or more pipeline rendering device in the set of pipeline rendering devices; each display device in the set of display devices is associated with a pipeline rendering device in the set of pipeline rendering devices; the host device selects a set of z-buffers from the rendered z-buffers using a CG camera definition from each pipeline rendering device in the set of pipeline rendering devices; each pipeline rendering device in the set of pipeline rendering devices uses a z-buffer received from the host device to render frame buffers and send the frame buffers to the associated display devices; and each display device uses the frame buffers to provide display view perspectives of a scene to one or more users.
 3. A distributed rendering and display system according to claim 2, wherein the at least one CG camera definition defines the position and orientation of the at least one CG camera that is used to render scene geometry.
 4. A distributed rendering and display system according to claim 2, wherein: the set of display devices are configured as a single display; and the display view perspective from each display device is visible on the single display. 